Transfer Learning for Multiagent Reinforcement Learning Systems

Reinforcement learning methods have successfully been applied to build autonomous agents that solve many sequential decision making problems. However, agents need a long time to learn a suitable policy, specially when multiple autonomous agents are in the environment. This research aims to propose a Transfer Learning (TL) framework to accelerate learning by exploiting two knowledge sources: (i) previously learned tasks; and (ii) advising from a more experienced agent. The definition of such framework requires answering several challenging research questions, including: How to abstract and represent knowledge, in order to allow generalization and posterior reuse?, How and when to transfer and receive knowledge in an efficient manner?, and How to evaluate the transfer quality in a Multiagent scenario?. 1 Context and Motivation Reinforcement Learning (RL) [Sutton and Barto, 1998] is an extensively used technique for autonomous agents with the ability to learn through experimentation. First an action that affects the environment is chosen, then the agent observes how much that action collaborated to the task completion through a reward function. An agent can learn how to optimally solve tasks by executing this procedure multiple times. The main limitation of RL is that agents take a long time to learn how to solve tasks. However, like in human learning, previous knowledge can greatly accelerate the learning of a new task. For example, it is easier to learn Spanish beforehand knowing Portuguese (or a similar language). Many RL domains can be treated as Multiagent Systems (MAS), in which multiple agents are acting in a shared environment. We are specially interested in Cooperative Multiagent RL (MARL), in which all agents work cooperatively to solve the same task. In such domains, other type of knowledge reuse is applicable. Agents can communicate to transfer learned behaviors. In the language learning example, being ⇤This research is supported by CNPq (grant 311608/2014-0) and São Paulo Research Foundation (FAPESP), grant 2015/16310-4 taught by a fluent speaker of the desired language can accelerate learning, because the teacher can identify learner’s mistakes and provide customized explanations and examples. However, learning how to actuate in a MAS may be a difficult task, since the environment becomes non-stationary due to the parallel actuation of multiple agents. Transfer Learning (TL) [Taylor and Stone, 2009] allows to reuse knowledge acquired in previous tasks, and has been used to accelerate learning in RL domains and alleviate scalability issues. In MARL, TL can either reuse knowledge from previously learned tasks or from agent communication, in which one agent can transfer learned behaviors to another agent. Even though TL has been used in many ways in MARL, there is no consensual answer to many aspects that must be defined in order to specify a TL algorithm. This research aims to specify a TL framework to allow knowledge reuse in multiagent domains from both previously learned tasks (when available) and agent tutoring, two scenarios that are common in human learning. 2 Research Goals and Expected Contributions This research aims to propose a Transfer Learning framework to allow knowledge reuse in Multiagent Reinforcement Learning, both from previous tasks and among agents. Specifying such method requires the definition of: (i) A model which allows knowledge generalization; (ii) What information is transferred through tasks or agents; and (iii) How to define when the knowledge of a given agent must be transferred to another. Figure 1 depicts the proposed framework. The agent extracts knowledge from advice given by other agents and previously solved tasks to accelerate the learning of a new task. The solution of this new task can then be abstracted and added to the knowledge base. 3 Background and Related Work Single-agent RL domains are usually modeled as a Markov Decision Process (MDP), which can be solved by RL. An MDP is described by the tuple hS,A, T,Ri [Puterman, 2005], where S is the set of environment states, A is the set of actions available to an agent, T is the transition function, and R is the reward function, which gives a feedback toward task completion. At each decision step, an agent observes the state s and chooses an action a (among the applicable ones Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16)